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A Guide for Evaluating a Reading or Language Arts Program 

Overview 

There are many reasons for evaluating educational outcomes. Some evaluations 
are guided by specific questions that rely primarily on traditional test scores: Is a 
particular reading program effective, for example. The overall reason for evaluating a 
reading or language arts program, however, is to find out how things are going: 

• Are there any changes that need to be made? 

• Is a stronger commitment needed to strengths identified in the program? 

• Are specific resources needed to make it more successful? 

This guide is about evaluation that can help you answer relevant questions about 
improving a reading or language arts program. Planning such an evaluation begins with 
some fairly broad considerations. If you want to conduct an evaluation that can help your 
program improve, it is imperative that you take a more inclusive look at it — beyond just 
reading the scores from tests you administer. 

This guide is organized into five sections: An overview section that introduces the 
evaluation approach is followed by a section for each of the steps in the evaluation 
process described in this first section. Several blackline masters are included at the end of 
the guide for you to use in getting your evaluation underway. These can be modified to 
match your specific needs. An annotated bibliography is also included to guide you to 
additional evaluation suggestions. 

There is nothing wrong with highly targeted test-score evaluations. An analysis of 
many evaluations of teaching reading and the language arts indicates that a majority of 
them were narrowly focused, and more often than not, they relied almost solely on test 
scores. They were not, however, what can be called program evaluations. Program 
evaluations go beyond scores and descriptively reveal strengths worth supporting and 
intensifying, and weaknesses to be eliminated. Most importantly, they can clearly 
indicate how to go about doing that. 

Wider-ranging evaluations that can inform good educational decision-making 
require a mix of different data types and some planning. Besides test scores from “right- 
answer” tests, an evaluation can gather rich information from a variety of other 
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evaluation methodologies, including alternative performance-based assessments, 
observations, surveys, and interviews. 

Attention to the audiences interested in your evaluation will help you decide what 
questions you want answered. A consideration of where you can get those answers will 
tell you which methodologies you can use. Many who are interested in the evaluation of 
your language arts or reading program will need the standardized or criterion-referenced 
test scores to be satisfied. But these data that they look on as “objective” can be enriched 
with a broader array of important and relevant information. Using a broad range of 
information is the only way that some important questions and issues can be addressed. 

This is particularly true in evaluating language arts and reading programs. They 
are designed to develop the ability to think and solve problems by using the skills and 
strategies of reading, writing, listening and speaking. We do this with some purpose for 
using the language — even if it is simply to enjoy ourselves. All good theory on the use 
and development of language tells us that it is interactive in this way. 

We want our programs to teach children to use language more effectively in valid 
and authentic ways that will serve them in and outside of school. Whether we are 
succeeding cannot be adequately described just with test scores. Language use, our 
educational goals, and the programs we use to achieve them tend to defy clean statistical 
description; and rich, more authentic data sources are needed to validly depict them. 

Gathering the kinds of information for your evaluation requires some initial 
thinking and brainstorming. The suggestions offered here for planning an effective 
evaluation might seem sequential, but actually they should be interwoven in a way that 
allows each decision you make in planning the evaluation to inform the others. It seems 
linear and is most neatly explained in that way here, but in reality, the general steps 
proposed are interactive — just like the language abilities we want to develop. 

These, then, are four major activities you can pursue in planning your evaluation: 
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Step 1: Clarifying what it is that you and others want to know: Rationalizing 
your study 

By brainstorming about what you want to learn, you can clarify and begin to 
organize the questions that you will attempt to answer. You will be considering who will 
want to know what; that is, identifying the audiences for your study and their needs. In a 
word, you will be rationalizing the study; and the result will be very helpful when you set 
out to explain to teachers, the public, and others why you are evaluating the program. 
Quite simply. Step 1 looks like this; 
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Clarify what it 
is that you 
and others 
want to know: 
rationalize 
your study. 



Basic activities : 

• Write questions about things 
you want to know. 

• Ask teachers, parents, others 
what they want to find out. 

• Group the questions. 



Who can do this best : 

You and your staff will 
probably want to do 
this, but audiences you 
identify will play an 
important role. 



The focus in this initial step is to target what needs to be learned and to state that 
in questions as clearly and simply as possible. Many evaluators prefer to structure these 
as decisions that they feel need to be made and by listing the options that they have in 
deciding. That more deductive approach tends to preclude discovering options as 
answers, but the inductive approach of more open-ended questioning can get somewhat 
unwieldy for one study. 

Even so, it is a good idea to be expansive and open in this step and to consider a 
full range of questions. Later, if you have too many questions for a single evaluation 
effort, you might need to select the most important questions in order to focus your 
efforts, or to plan several studies. Or — since evaluation is a normal task for educators — 
you may be seeking answers to good questions about your program on a continual basis. 

Generating Good Questions 

Begin by brainstorming your own questions, which are apt to begin with, and 
grow out of, the first: “How well are we doing with our language arts/reading program?” 
Then talk to your teachers, parents, and other audiences who will be interested in the 
information you produce. Find out what their questions are and merge them with your 
list. 
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Below are some samples of the kinds of questions that should focus your study. 

Most of the questions will fall into several groups similar to the four groups below. 

General Questions about Language Performance and Reading Habits 

• How well are our students learning? How well do they read and write? 

• Do our students improve significantly in our program? Is our investment in the 
program paying dividends? Are we getting our tax dollar’s worth? Is our investment 
in the program paying reasonable dividends? 

• How much and how often do our students read and write? What kinds of reading and 
writing do they do? Do they use reading and writing effectively throughout the 
curriculum? Are we motivating them to become more effective language users? 

• How much of our students’ language development takes place in school? How much 
in the classroom? How much at home? Is there any indication that parents are aware 
of language growth? Do they report increased reading at home? 

• How heavily used are our school libraries? What kinds of language use are indicated 
by the use made of our libraries? 

• How interactive are reading and writing and the other language behaviors in our 
instruction? For example, does reading instruction generate student writing? 



Availability of Materials 

• Are there adequate instructional materials and free reading materials available to our 
students? Do teachers and students have the support materials they need? How 
extensively are the materials that are available used? 

• Do our school libraries have adequate copies of the trade book selections 
recommended in the program we are using? 

• Do our schools provide the resources to integrate the language arts program with the 
music, art, and other activities suggested in the program? Is it relatively easy for the 
teachers and students to follow up with activities suggested by and selected from the 
program? 



Evaluating a reading or language arts program 

Teacher/Instructional Influences and Concerns 

• Have our teachers been adequately introduced to teaching concepts and techniques 
that could make the program more effective? How do the in-service opportunities we 
offer tie into the program? 

• Have the teacher resource materials accompanying the program been made fully 
available to the teachers? How frequently are they used? 

• What opportunities do our teachers have to consult about problems and to ask 
questions related to the program? 

• What kinds of assessments are used in our schools and classrooms? How closely and 
in what ways is assessment of language development linked to instruction? What 
assessments provided by the program are made available to and used by our teachers? 
Is any emphasis we are placing on alternative assessment giving teachers a better 
handle on instructional decision-making? 

• How well does individual teaching fit into the stated goals of our curriculum and the 
program? How are these goals structured across grades? How much attention do our 
teachers pay to this structuring? What influence have teachers had on setting program 
goals? Do they feel involved? 



Parental Concerns and Influences 

• What evidence is there of an increased parental role in the language development of 
our students? What can be identified as contributing to the impact of parents? Are 
parents of our students reading to their sons and daughters? Do they read things that 
their children write? Do parent-child discussions of the ideas involved in what 
students read and write take place with much frequency? 

• Are materials available in the program provided to teachers, and how often do they 
use them? What kinds of information about the language arts/reading program and the 
progress of their children do parents receive? Do they, for example, get reading lists 
appropriate for their sons and daughters? 



ERIC 



Page 5 g 



Evaluating a reading or language arts program 



• What opportunities has our program provided for parents to learn about the language 
development of their children? Do they get to contribute to a better understanding of 
that development? 

It is important to re-emphasize that these are merely samples of the kinds of 
questions you may consider in Step 1. This is neither a complete list of the questions you 
could articulate nor a checklist to be applied. Your brainstormed list may be longer than 
the one above, which is probably longer than the final list you will use to focus your 
study. Your initial list of questions can be pared down to focus on the most relevant and 
important. Five or six key questions may be all that you want to pursue in an initial 
evaluation, but don’t limit the brainstorming as you get started. Let the list get long. You 
can synthesize, eliminate redundancies, and cut back later. 

You may decide that some of the questions above are important and match the 
needs of your staff and/or administration. You may have many others that are different 
and relative to your schools and your program. You may realize already that the study 
you want to conduct cannot answer ever)1;hing that you might be interested in asking 
about your program. Still, free-wheeling and grouping questions in the list is what leads 
you to that decision and can help you prioritize and select the most important ones and 
ones that you can reasonably answer. 

As you do this, you will see reasons to combine some of them. You may discover 
questions you feel you have misplaced in the wrong category. You may identify 
questions that are essentially the same that appear in two different general categories you 
have created and decide to seek the answer under only one so that the data which 
determine it will not have to be combined later. You may decide to rework your 
categories and your organization, so it becomes more evident how Steps 1 and 2 are 
interrelated. 

Remember that the sample questions above are ones that could grow out of 
considering the information needs of a variety of audiences. The table below 
demonstrates how just a few of the questions can serve those needs. 
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Audience 


Pertinent Question 


Possible Applications of the Answer 


Board of 

Education 

Media 

Administrators 

Legislators 


Are our schools 
effective? Are our 
children reading better? 
Is society getting its tax 
dollar’s worth? 


Increase funds to support identified 
strengths, to shore up weak spots; review 
curriculum; raise teacher qualifications; 
become better informed about programs 
and their objectives. 


General Public 


How well do our 
children read? How 
much do they read? 


Promote library collections/circulation; 
read more to children at home; support 
reading activities in school; pressure for 
more teacher aides. 


Principals 


Do reading and 
language instruction in 
my school contribute to 
making better citizens? 


Promote activities that develop thinking 
and articulation of social consciousness. 
Recognize articulate students; create 
opportunities for communication on real 
issues across grades, between schools, 
between home and school. 


Teachers 


What aspects of my 
instruction are deemed 
most successful? 


Create more opportunities in the program 
to apply teacher strengths; seek more 
support in time and materials; challenge 
and perfect other techniques that seem 
less productive; promote an awareness of 
other activities that seem under- 
appreciated; lobby for more opportunities 
for student experience with reading and 
language. 


Parents 


Is my child going to be 
more successful 
because of current 
instruction? 


Press for instruction with applications to 
the student’s life. Seek advice on how to 
enlarge opportunities at home. 



Relating Your Questions to Decisions to Be Made 

Note that the question samples listed above can grow out of decisions that you 
have already identified as ones that need to be made. The table below gives samples of 
how that can work. 
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Decision to be made 


Relevant question 


You must decide whether to use a 
performance assessment in addition 
to — or in lieu of — one of the 
standardized tests you regularly 
administer. 


How well do performance assessments 
inform our teaching? How well do 
audiences interested in accountability 
accept those results? Which published 
tests are good matches for the program 
materials we use? 


You must decide whether to fund and 
budget system-wide an expansion of a 
volunteer tutorial program that has 
been piloted at a few schools. 


Has the experimental program had any 
discernable impact on the reading and 
writing behavior of the students in the 
pilot schools? How well did it mesh with 
the regular program and its materials? 


There is pressure from a group of 
teachers to incorporate portfolio 
assessment into your accountability 
formula; another group opposes this, 
saying it will eliminate the students’ 
sense of portfolio ownership. 


How are our teachers using portfolios? 
How many use them? How does the 
methodology serve their instruction? 
How different would the portfolios their 
students keep be if they were 
submitted to evaluators? How would 
that impact portfolio use in the 
classroom? Is there a way to meet both 
needs? 


There is a request to fund a desktop 
publishing system for each school in 
the district. 


What impact are classroom 
publications having on student 
language development? 



With the answers to the questions relevant to the decision to be made, you can 
line out your options more completely and clearly than you could have expressed them 
before asking the question. 

BEST COPY AVAILABLE 
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Step 2: Deciding where and how to get the information: Targeting the data 

The way that the four steps for planning and conducting an evaluation are 
interlocked and interact becomes clearer at this point. Step 2 involves deciding where you 
can get the information and how to go after it. You will soon discover that many of your 
audiences are your sources as well. Some of your sources are, in fact, the assessments 
themselves. And having determined whom to ask and how to ask it, you have actually 
done a major part of Step 3: Implementing the study. The procedure involved in Step 2 
looks like this: 
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Deciding 


Basic activities: 


Who can do this best: 


T 


where and 


• Identify your sources: Who has 


Again you and your staff 


E 


how to get 


the information to answer 


may need to make 
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the 


particular questions you want to 


these decisions, but 


■ 


information: 


ask? 


consult with your 


2 


targeting the 


• Select the assessments you 


sources and with 




data 


will use (the ways you will 
collect information). 


experts if need be. 



The two major considerations of this step flow quite logically. 

Identifying Your Sources 

Take a look at each question you would like to ask in your evaluation. Who has 
the information to answer it? Many questions you may articulate for an evaluation can be 
answered by more than one source, and when that is the case, a good evaluation will 
seldom rely on just one. In general, the sources that can inform your questions are fairly 
obvious. The planning table below exemplifies this using only a few questions to make 
the point. You can create a similar, more complete table for the questions selected for 
your evaluation study. 

When you are finished, you should regroup your questions by probable data 
sources. This will help , you witl]i|he second major activity in Step 2. With each question 
is a shorter tag that will be used in a following table to illustrate how question/source 
combinations are matched to types of information gathering and information. 
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Possible Questions 


Information Source 


How much reading are our students doing outside of 
school? 

(Outside Reading) 


Students, parents, 
librarians, portfolios 


How receptive are different evaluation audiences of 
portfolio assessment? 

(Portfolio Acceptance) 


Media representatives, 
administrators, 
parents, education 
critics 


How well does the performance of our students 
compare to students in other school systems? 
(Comparisons) 


Norm-referenced tests 
/ performance 
assessment 


How well are our students meeting state and local 
standards? 

(Competency) 


Criterion-referenced 
tests / performance 
assessment 


How well do our students apply what they read? 
(Application) 


Performance 
assessment: teachers; 
parents 


How aware are our students of their own language use 
and development? (Self-assessment) 


Portfolio assessment; 
teachers: students; 
parents 


How competent are our students with language 
mechanics? (Mechanics) 


Norm-referenced and 
criterion-referenced 
tests; portfolios: 
performance 
assessment: teachers 


What do our students read by choice? 
(Reading Choice) 


Students, parents, 
librarians, teachers 



Selecting the Assessments (the Ways You Will Collect Information) 

The assessments that you use to collect the information are extremely important. 
Don’t make the mistake of concluding that one assessment will do the job. Neither should 
you administer a variety of assessments just for the sake of variety. Each assessment must 
be selected to the extent that it provides the information you need. 

In selecting assessments remember that an assessment is not necessarily a test. An 
assessment is a means of collecting information. Interviews, surveys, observations, and 
reviews of materials are all considered assessments because they are each a means of 
collecting information. 

14 
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The list below provides some idea of the wide variety of information collection 

strategies and assessments that are available to you. Check the references at the end of 

this booklet for explanations of other assessments. 

• Norm-referenced assessments: These tests have been standardized, and the scores 
are used to compare students to those in the norming population. Norm-referenced 
tests are useful as general indicators of achievement. Most norm-referenced tests use 
multiple-choice items. However, a number of nationally standardized tests have now 
added student written responses. 

• Criterion-referenced assessments: These tests often look very similar to norm- 
referenced tests. However, they are more closely focused on the objectives used to 
design the tests. Criterion-referenced tests are used to determine how well students 
have achieved specific objectives-without concern about how other students perform. 
The focus on comparison to a criterion rather than other students (norms) is the 
primary difference between norm-referenced and criterion-referenced tests. 

• Writing assessments: Many states and school districts administer writing 
assessments in which students are asked to write in response to a given problem, 
topic, or even picture. Some writing assessments allow students to engage in pre- 
writing and revising while others ask students to produce just a single draft. Scoring 
of writing assessments takes several forms including single scores or diagnostic 
scores of various aspects of writing as mechanics and content. There are no norm- 
referenced writing assessments. All of the writing assessments are compared to 
anchor papers or model responses. Thus, all writing assessments would fall under the 
more general category of criterion-referenced assessment. 

• Performance assessments: Performance assessments take many forms and are 
always criterion referenced. The major focus of most performance assessments is the 
application of knowledge. The goal is to determine if a student can solve a problem 
by using what he or she knows. 

• Portfolio assessments: Portfolios are used in the classroom to encourage students to 
collect work samples so they can talk with their teachers about ways of improving. 
Show portfolios are selected samples of work completed over a period of time to 
demonstrate student growth and development. 




ERIC 



Page 1 1 




Evaluating a reading or language arts program 

• Work samples: All kinds of samples of student activities fall under this category. 
Work samples are collected in much the same way as portfolios. Indeed, portfolios 
could be considered collections of work samples. 

• Interviews: Interviews, especially those that are more loosely structured, allow for a 
discussion of a range of topics. The interactive nature of an interview allows issues 
and concerns to develop rather than merely responding to a structured list, as would 
be the case with a survey or questionnaire. 

• Surveys and Questionnaires: These provide a means to collect information and 
reactions on specific topics from a large sample. Open-ended responses can be 
included on a questionnaire or survey, but most of them are structured by a list of 
issues. Surveys and questionnaires look simple to develop and use. However, the 
development of a survey questionnaire that will produce valid results is a difficult 
task. 

• Observations: Observations can be structured or unstructured, but they should be 
focused by issues or concerns. Classroom observations should be more than merely 
watching a class or teacher. Observations in a classroom, school library, or hallway 
can provide useful information about reading habits and attitudes. Good observations 
need to be planned and discussed so that those being observed do not believe they are 
a means of checking-up. 

• Checklists: Checklists can be used as a means of quickly and efficiently recording 
information during an interview or observation. 

• Others: There are a number of data collection procedures to determine reading 
levels, writing strategies, and language skill development. These include such 
assessments as running records, oral think-along assessments, retrospective 
discussions of writing strategies, reading miscue analyses, and cloze tests. 

The above list should not be considered inclusive, nor should you attempt to use 
as many of these approaches as possible. The list is provided merely to remind you that 
there are numerous ways to collect information— numerous ways to assess. The selection 
of specific assessments should be- determined by the questions in your evaluation plans 
and the information you think you need to answer those questions. 
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The form of assessment used can be determined by a number of factors. 
Obviously, the question you are asking is the primary one; and the sources you have 
identified for answering it also suggest which ways you can gather your information. 

The match, then, of question/source combinations to assessment types can be 
considered in planning as it is in the next table, in which the tags for the question/source 
combination refer to the previous table where the questions are matched to possible 
sources for their answers. 



Question/Source Combination 


Evaluation Assessments That Would Serve 


Outside Reading/Students, 
Parents, Librarians, Portfolios 


Surveys, Interviews, Portfolio Analysis, 
Observation 


Portfolio Acceptance/Media 
representatives, Administrators, 
Parents, Education Critics 


Interviews, Surveys, Educator Observation 


Comparisons/[tests/student 

performance] 


Norm-referenced, Criterion-referenced, and/or 
Performance Assessments 


Competency/[tests/student 

performance] 


Criterion-referenced, Performance 
Assessments; Show Portfolio Assessment; 
Writing Assignments 


Application/[tests], Teachers, 
Students 


Performance Assessments, Interviews 
(including teacher Observation), Surveys, 
Portfolio Assessment, Writing Assignments, 
Think-Alongs 


Self-assessment/Portfolios, 
Teachers, Students, Parents 


Portfolio Assessment, Interviews, Surveys, 
Observation, Think-Alongs 


Mechanics/[tests], Portfolios, 
Performance Assessment, 
Teachers 


Norm- or Criterion-referenced Assessments, 
Portfolio Assessment, Performance 
Assessments, Interviews, Writing 
Assignments 


Reading Choice/Students, 
Parents, Librarians, Teachers 


Surveys, Interviews (about Observation), 
Check Lists 





17 



Page 13 



Evaluating a reading or language arts program 



Step 3: Designing and conducting the study: impiementing the evaiuation 

Actually, you have been designing the study from the time you began thinking 
about it. What remains is to select from the possibilities you have identified: Which 
questions do you intend to ask? What sources will you rely on for the information? What 
assessment types will you use? In effect, you are polishing, specifying, and refining the 
work you have done in Steps 1 and 2. 

There are other organizational details that may be important, depending on the 
scope of your evaluation. You can set a schedule to accomplish everything you will do 
and assign responsibilities for each task within some chain of command. You will need to 
acquire or develop the instruments you need, and you may decide to pilot your design and 
any instruments you have created, revising if and as the piloting recommends. 

Then, of course, comes the heart of the matter — the job you have been getting 
ready to do; gathering, synthesizing, and interpreting the data collected to get at answers 
to your questions. Step 3 represents the bulk of your evaluation, and if you have 
completed the first two steps effectively, this one should go smoothly. Step 3 looks like 
this: 
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Designing and 
conducting 
the study: 
impiementing 
the evaluation 



Basic activities : 

• Specifying questions, sources, 
and assessments. 

• Scheduling the various parts of 
your evaluation. 

• Assigning responsibility for 
particular tasks and aspects of 
the study. 

• Acquiring or creating the 
assessments you need. 

• Piloting the plan and any 
assessments created; making 
any necessary revisions. 

• Gathering the information. 

• Synthesizing and interpreting 

the data gathered. 



Who can do this best : 

You and your staff 
(including assessment 
specialists) and then the 
teachers and other 
volunteers who are 
assigned tasks, and 
often a particularly hard- 
working and dedicated 
committee to synthesize 
and interpret. 
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Specifying Questions, Sources, and Assessments 

If your planning in Steps 1 and 2 has not led you to specifying which particular 
questions you want answered, who can give you those answers, and which assessments 
you can use to get them — that needs to be done before beginning the collection of data. 
The next table uses just three sample specific evaluation questions to illustrate the options 
you have for creating a rich database. In an evaluation plan at this stage, these questions 
would be grouped by source so that a single instrument — a survey administered to 
parents, for example — could gather information on numerous questions you want to ask 
parents. And some questions will be repeated across sources to acquire various available 
perspectives. 



Initial Evaluation Plan for a School or School District 
Focused on Three Possible Evaluation Questions 


Question 


Source 


Assessment 
(How to Collect) 


Typical Tasks 


Who Does It 


1 . How aware 
are parents of 
the goals of 
our program? 
How well do 
our goals 
match their 
concerns? 


Parents, 

administrators, 

teachers 


Survey, 

interview, 

questionnaire 


Develop/write the 

questionnaire. 

Select parent 

sample to be 

surveyed. 

Administer 

questionnaire 


Staff with the 
assistance of 
a volunteer 
parent 
committee 


2. What kind 
of reading do 
our students 
do? 


Librarians, 

parents, 

teachers, 

students 


Interviews, 

surveys, 

questionnaire 


Write protocol to 
guide interviewers. 
Identify interviewers. 
Select people to be 
interviewed. 

Conduct interviews. 
Synthesize results 


Staff, 

teachers, 

volunteers, 

librarians, 

parents 


3. How well 
do our 
children 
write? 


Writing test, 
classroom work 
samples, show 
portfolios, 
classroom 
observations, 
performance 
assessment that 
includes writing 


Selected writing 
test, portfolios, 
obsen/ations 


Identify or create the 
writing assessment. 
Discuss work 
sample collections 
with teachers. 
Classroom 
obsen/ations. 
Scoring the writing 
tests. Analyzing 
work samples 


Teachers, 

teacher 

committee 
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This table, which moves from planning guideline toward implementation guide, 
begins to lay out the actual plans you would need to develop. 

Note these other things about this example: 

• A survey questionnaire administered to parents could easily seek answers to other 
questions you have, as long as it does not overburden the parent filling it out. 

• The answer to question 2 would probably be sought from other sources, as well as 
from librarians. Some of those, such as parents, might be surveyed. Library records 
might be an excellent source as well. Remember that the table above is only an 
example to demonstrate how you can go about fleshing out your plans for your 
evaluation. 

• Another source of the answer to question 3 could be the parent and/or student 
surveys. Students are usually fair and honest in their appraisal of such things as the 
amount and kind of writing they are doing. Some students do a great deal of writing 
outside of school, often associated with clubs and hobbies. Others do very little. Your 
evaluation will be richer if you include these sources of information. Your particular 
program and situation will probably add a considerable amount of detail to this brief 
example. 

Scheduling the Various Parts of Your Evaluation 

At this point, or even before, you may have a time schedule of what will happen 
in your evaluation. It helps to have a master plan laid on a timetable that estimates how 
long each aspect of the study will take and attempts to schedule aspects of the study so 
that they fit together within a reasonable time frame. How long will each part of the study 
take? When should it be started so that the results are ready in time to be synthesized with 
the results of the other parts and to be interpreted for reporting? 

Your study may answer a few vital questions in a matter of weeks, or it may be a 
more extensive evaluation that takes a longer period of time. Although it is hypothetical, 
and treats only a few aspects that an evaluation may include, the following chart attempts 
to demonstrate one way such a timetable can be structured. 

Suppose, for the sake of illustration, that you have spent a spring semester and the 
summer planning your study, and your goal is to implement it in the following school 
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year. You would then conduct your analysis and synthesis during the following summer. 
The scheduling of the implementation might go something like the one below. The sraph 
is not offered as a model but is meant only to demonstrate how such an overall time 
framework might help keep your study on schedule. 

Your situation will dictate the exact timetable for your evaluation. It could easily 
take a full year if you decide to include a lot of your questions. However, by selecting a 
few questions to answer, your evaluation may require only a month or two. Using a 
visual aid like this timetable, however, allows everyone to track the progress of the 
evaluation. 



Month 

Aug. Sep. Oct. Nov. Dec. Jan. Feb. Mar. Apr. May 

Pilot instruments; 
finalize plans; acquire 
all needed materials 

Assist teachers of students who 

will submit show portfolios 

Train interviewers 

Begin observations of 

selected classrooms 

Administer norm- 
referenced tests. Begin 
teacher and librarian 
interviews 





Begin teacher and parent 




surveys and interviews. 


[> 


Administer performance 


o 


assessments 


< 


Collect 




portfolios 



Administer 

criterion- 

referenced 

assessments 

Intensify analysis of 

incoming data 

Intensify synthesis of 
different data types 

Continue analysis. 
Focus on 

interpretation of data 
synthesis 



Evaluating a reading or language arts program 



Assigning Responsibility for Particular Tasks 

A clean operation always depends on everyone in a project knowing who is 
responsible for which task. You should know by name, each staff person, teacher, or 
volunteer who is to oversee each aspect of your study. The smaller the study is, of course, 
the more informal that arrangement can be. Larger evaluation efforts — particularly those 
that will be producing a variety of data types to be synthesized in order to answer some 
important questions — can profit from a chart or list of tasks with the names of who is 
responsible for each. Such organizing resources usually show whom each person reports 
to, so that information, questions, and problems can be channeled efficiently. 

Who, for example, will gather together all of the data from different tests given at 
different times and at different grade levels? Who helps that person schedule the 
administration of the tests? At each school where testing is to be done, who will oversee 
it? Who will help the person responsible for the rating and analysis of show portfolios? 
Who will help the person overseeing the gathering of criterion-referenced test data 
compare and synthesize their findings? 

Some responsibility charts are effectively combined with schedules and show just 
when particular participants in your study will be expected to play their roles. 

Acquiring or Creating Needed Assessments 

Basically this aspect of Step 3 is one of making sure that all the materials that 
participants of your study will need are available when they need them. Are there enough 
copies of the particular levels of tests that you will need on hand? Have you decided on 
assessments that must be created or modified from ones used in the past? 

This aspect of your planning should also look at other ways of gathering 
information that you have decided to use. Are there guidelines and protocols to help 
people who will be doing interviewing and observing, so that what is gathered by 
different interviewers can be easily coordinated? Are the guidelines for the preparation of 
show portfolios clear and available to all teachers who will take part in that data 
collection? How about the guidelines for analyzing the portfolios or for rating 
performance assessments? 
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Finally, do you want to consider training sessions for any of the participants in 
your study to help assure that it is implemented evenly and effectively? 

Piloting and Revising Your Plans and Assessments 

The scope of evaluations can vary greatly. Yours need not be large to be valuable, 
and implementing whatever you do can be made a part of ongoing management. This can 
simplify concerns about responsibilities, for example, and it can simplify scheduling. If, 
however, the evaluation you are planning is larger, you might be interested in piloting 
aspects of your plans and trying out the ways of gathering information that you have 
selected. 

Are the directions and protocols you have prepared for a team of observers going 
to generate the collection of similar data from each of them? Are the questions that have 
been prepared for interviewing parents or librarians effective? Will the team of volunteers 
rating performance assessments be able to follow the rubrics and anchor samples you 
give them and come up with reliable results? 

This kind of piloting takes time — especially if you allow for revisions in your 
plans or assessments or procedures that the piloting recommends. So do not overburden 
yourself with this concern. If you can get it done, however, it makes sense to find out 
how good your plans and assessments are and how well they will work. Often this all- 
important step results in clarification and simplification that will make the data you 
collect far more useful, dependable, reliable, and valid. 

Gathering and Synthesizing the Information 

“Finally!” you may be inclined to exclaim. You are ready to begin your 
evaluation! But everyone who has been involved in designing and planning it will 
understand — with a conviction that can only come from involvement — that it has been 
underway for some time; Planning is an essential part of the study. 

If you have planned and organized adequately, the administration of the various 
aspects of your evaluation should go according to your carefully sequenced time plan; 
and the arrival of the data should find its way to whoever is to synthesize and interpret it. 

That task, again, is accomplished differently depending on the scope of your 
study. A single researcher, for example, could conceivably do it. Depending on how 
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high-stake the evaluation is — how many decisions will be made using its results — you 
may have arranged for the involvement of more individuals in this aspect of the study. 
You may have included representatives of different audiences as part of the team. 

With the data in hand, whoever is doing your data analysis should go through the 
questions that survived to the final implementation and articulate answers where that is 
possible. The predominant ones are apt to be: 

• How are we doing? 

• What changes need to be made? 

• What commitments need strengthening? 

• Do we have the resources we need to improve? 

You, your staff assigned to this task, volunteer committees, or whoever is doing 
this analysis should not force answers through questionable interpretations and need not 
fear the need to qualify findings in order to assure their credibility. 

Ideally you will have outstanding and highly qualified individuals involved to 
summarize the answers as a descriptive report of the evaluation of your language arts or 
reading program. Accompanying this should be a set of recommendations that grows 
clearly out of the synthesized findings. It may be necessary — to maintain full 
credibility — to have a secondary set of findings and recommendations that seem 
somewhat “apparent” but are not endorsed by the study without any reservations. 

Have ready a process of discussing and reviewing the findings of your study. Let 
those who conducted the analysis consider any challenges and recommendations for 
changes that emerge. The final task in the analysis should be to recommend a format for 
the final report on the evaluation to be written by competent staff persons. 
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Step 4: Reporting to the people who need to know what you have found: 
Satisfying the audiences 

You have come to the point where you can report what you have learned about 
your program. You should be able to answer the predominant questions, as well as 
questions that are pertinent to particular audiences. You can address them all in a general 
report and plan to write specific reports with descriptions and recommendations for 
specific audiences, such as teachers, parents, librarians, and the students themselves. 

The final stages of Step 3 have indicated how you can get to this point using 
committees, consultants, or whomever you think should be involved. It is worth stressing 
that it is a good idea to have members of the different audiences represented in the 
process, but the writing of what has been found does not have to done by committee. It 
can be delegated to a competent staff or volunteer writer, whose efforts are guided and 
carefully reviewed by those who hand analyzed the data and made the recommendations. 



s 


Reporting to 


Basic activities: 


Who can do this best: 


T 


the people 


• Outlining your findings. 


The committees or 


E 


who need to 


• Drafting an analysis of the 


other analysts who have 


P 


know what 
you have 


whole database, reporting to 
particular audiences, and 


synthesized and 
interpreted the data 


4 


found: 
satisfying 
the needs of 
the 

evaluation 

audiences 


making recommendations. 

• Distributing a draft of the report 
so members of different 
audiences can react to it, and 
then revising and polishing it. 

• Making sure that your report is 
widely circulated and properly 
understood. 


may select members to 
work as a writing team. 
Such a group may want 
to work with report- 
writing specialists on 
your staff, who will be 
responsible for the 
actual writing. Make 
sure you are prepared 
to offer assistance in 
the use of the 
document. 



Outlining Your Findings 



It always helps to have a sense of the whole — to be able to consider the different 
aspects of what you have found in an evaluation from a broader perspective that helps 
you keep a balance and to better understand how different findings may relate. The 
individuals who have been involved in synthesizing and analyzing the data have no doubt 
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already done that as they interpreted it and related it to the study’s questions. They should 
play a major role in preparing the report. Having a better grasp on the data than anyone 
else, it should be relatively easy for them to outline what has been found. That outline 
may closely resemble the articulation and final organization of the questions before 
beginning data collection, or it might demonstrate how an inductive gathering of data 
information has provided new insight. 

No attempt to exemplify how this may develop is suitable here. Your particular 
educational setting, the study you designed and the way it was designed, the questions 
you set out to answer, and other factors should make the emphases of your report unique. 
If one could predict even broadly what it would say, the evaluation would not be 
necessary. It is important, however, to refer back to the original questions that guided the 
evaluation. 

Drafting a Broader Analysis, Reporting to Particular Audiences, and Making 
Recommendations 

Ideally, the results of your study will support some interesting generalizations 
about your language arts or reading program. The first part of the report should explain 
these and be written for a general audience. These are the findings that are most apt to be 
summarized in the press, and that is appropriate because they should be articulated to 
reflect the balance of what was revealed across grade levels, schools, language behaviors, 
or other different units of the study. This does not mean that this part of the report will 
report no specifics about the findings. Rather, those kind of supportive and illustrative 
details are essential in each part of the report. 

Reporting what can be generalized does not mean dwelling on innocuous 
generalizations. What you find out about your program from a broad perspective might 
be quite involved, reassuring, or disturbing. A certain number of the questions you 
pursued will relate to this wider scope, and what you found could be discussed in this 
section. 

After taking this broad perspective, the study can treat what was learned in 
answering the questions that are of the most intense interest to particular audiences, like 
parents, then teachers, and the students themselves. These vital or vested particular 
interests may not have been obvious from the start of the study when the questions were 
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first grouped. They may have become obvious as the information-gathering procedures 
involving members of an audience proceeded. 

Finally, the report should culminate — either overall — or section by section in 
meaningful and useful conclusions and recommendations that, acted upon, can bring 
about change and strengthen your program. Note that this does not necessarily mean 
identifying weaknesses, recommendations might very well call for more commitments to 
certain identified strengths. 

With all of these suggestions, it is important to keep in mind that your audiences 
will be relying on the synthesis and analysis you have done. So your report should not be 
extensively long. It needs to focus on the key findings and contain succinct summaries. In 
short, it should be reader-friendly and easy for your audiences to understand. 

Distributing a Draft for Reactions in Order to Revise 

Once your report is finalized for distribution, you will want it to be 
enthusiastically received and treated as a reliable guide to improve your language arts or 
reading program. You have tried to ensure this trust in and acceptance of your study if 
you have involved members of all audiences in the evaluation process and by reporting 
for particular audience perspectives. 

But there is a final step that will help assure this. Distribute the draft of the report 
(clearly and distinctively marked “DRAFT”) to individuals in each audience who are 
willing to react to it. This should give you an indication of spots in the report that need to 
be clarified, or reworded. It could lead you to reconsider the appropriateness of the way 
the study data has been treated in the report, the validity of a recommendation, or 
whatever. The point of doing this is not to force changes on the findings but to make sure 
that the report of them is clear and fairly interpreted. The reason for doing it in draft form 
is to allow time to revise for accuracy and clarity. 

Making Sure the Report Is Widely Circulated and Understood 

The final draft of the report can have little impact beyond that the evaluation 
study itself had on participants unless it is read. It would be a shame to go through all the 
responsible planning and the careful implementation of the study and then fail to see that 
it is widely distributed in an attractive and engaging format. This involves more than 
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designing effectively and printing lots of copies. How will people get them? Equally 
important, how will members of your evaluation audience know that the study was 
conducted, that the report is ready and available, and that it holds something of potential 
interest to many of them? 

Obviously you will need to publicize the report by making sure that the media 
knows about it. You can prepare special press releases which are, in effect, carefully and 
crisply written summaries to facilitate media attention. You can prepare press releases 
about the findings as a whole, and about other more specific findings and 
recommendations. You can distribute the findings in abbreviated form that catches one s 
interest and whets an interest in the full report. You, personally, may want to take 
numerous copies of it to the schools and to meetings of teachers and parents. You can 
encourage groups with potential interest in your program to hold discussion groups based 
on the report. 

Finally, you can anticipate questions and reactions that people will have to the 
report and prepare extended analyses for personal presentations. Your staff and a 
volunteer group could prepare to do this. If you are not able to anticipate reactions, listen 
for them and identify aspects of the study and/or report that can be clarified by joining 
teachers, parents, or other interested parties in their discussion of your study’s findings. 
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An evaluation of 



will be conducted in the near future. The study will produce 
information that can help us strengthen the program and 
develop better language users in our schools. In planning for 
this study, we will seek the answers to questions of direct importance to teachers, 
librarians, parents, and students. 

Before settling on what those questions will be, we are seeking your input! 

Please take time to think about this project and contribute to its success. What 
questions do you think need to be asked? Please make a note for each question that 
explains why getting the answer to that question would be important to you. Thank 
you very much for your help. 





1 believe this question is important: 


The answer is important because: 


? 


! 


? 


! 


? 


! 



[name] 



I am a 
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An evaluation of 



will be conducted in the near future. The study will produce information 
that can help us strengthen the program and develop better language 
users in our schools. In planning for this study, we will seek the answers to questions 
about the way we are teaching language arts and reading. 

l^e hope to get a great deal of important information from teachers, librarians, 
parents, and students! 

Please take time to think about this project and contribute to its success. What 
information do you have that will help reveal how well we are doing? Please make a 
note for each question that explains how you think that information might be 
collected. 





1 have this kind of information: 


This is how you should collect it: 







I am a 

My name is 

You can contact me by 
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A General Finding: 


Recommendation: 


Rank 

1-3 





















Here are things we learned to report to particular audiences: 


Order to use in 
reporting 


What we learned: 


Audiences: 
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